Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Monitoring] Ensure Elasticsearch parts of Stack Monitoring work with metricbeat-* indices #90838

Closed

Conversation

chrisronline
Copy link
Contributor

@chrisronline chrisronline commented Feb 9, 2021

Relates to #73864
Relates to elastic/beats#24427

This PR is meant to update all existing Stack Monitoring UI code to correctly work off metricbeat-* and .monitoring-* indices for Elasticsearch-related areas of code. It is an extension of #88212 as it identifies the fields missed by actually using a working Metricbeat module (previously, we had to mock the output of the module which was error-prone).

This effort is hard to guarantee 100% completion because we need to ensure every part of the front-end works properly with both data sets. We have two main ways to test this:

  1. Spot-check manually that all charts/data values are properly presented in the UI
  2. Leverage our existing functional test suites

The main issue with 2 is that our existing functional test suites use archive data which is in .monitoring-* indices so they aren't very helpful in ensuring the metricbeat-* data path works (but are helpful in determining the .monitoring-* data path still does work) so I did some work to build archive data using metricbeat-* indices.

This work was done in several parts:

Part 1: Write a script that identifies all known aliases that currently exist in the metricbeat-* indices and write it to a file: https://gist.github.com/chrisronline/d24ec862efdb0cfa331c7ef785241116

Part 2: Modify the rebuild all function of the es_archiver script to build out metricbeat-* indices from the existing .monitoring-* data archives, using the known aliases as well as some custom logic: https://github.com/elastic/kibana/pull/90838/files#diff-b0251a00b48f25e70b98d6741ae88948b22f9220a5ba5914aaaab465038da659 (This script is ran like: node ../scripts/es_archiver.js rebuild-all)

Part 3: Duplicate existing test suites and use the newly created archives based off metricbeat-* indices: https://github.com/elastic/kibana/pull/90838/files#diff-36d8e173c4e5a530dd9d5d68faca3cfd3f69f3803f80efda74c1f8bc562f033b

Note: elastic/elasticsearch#71233 needs to be merged before all functional test will consistently pass

@chrisronline chrisronline changed the title [Monitoring] [Monitoring] Ensure Elasticsearch parts of Stack Monitoring work with metricbeat-* indices Mar 1, 2021
@chrisronline
Copy link
Contributor Author

Closing in favor of a feature branch in the elastic repo: #96205

@kibanamachine
Copy link
Contributor

kibanamachine commented Apr 5, 2021

💔 Build Failed

Failed CI Steps


Test Failures

Kibana Pipeline / jest / Jest Tests.x-pack/plugins/monitoring/server/lib/metrics.Metrics should export metric objects that match a snapshot

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches


Stack Trace

Error: expect(received).toMatchSnapshot()

Snapshot name: `Metrics should export metric objects that match a snapshot 1`

- Snapshot  - 0
+ Received  + 1

@@ -3350,10 +3350,11 @@
            "unit": "1s",
          },
        },
      },
      "app": "logstash",
+     "debug": true,
      "derivative": true,
      "description": "Number of events received per second by all Logstash nodes at the inputs stage.",
      "field": "logstash_stats.events.in",
      "format": "0,0.[00]",
      "label": "Events Received Rate",
    at Object.<anonymous> (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/x-pack/plugins/monitoring/server/lib/metrics/metrics.test.js:12:21)
    at Promise.then.completed (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:276:28)
    at new Promise (<anonymous>)
    at callAsyncCircusFn (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/utils.js:216:10)
    at _callCircusTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:212:40)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at _runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:149:3)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:63:9)
    at _runTestsForDescribeBlock (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:57:9)
    at run (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/run.js:25:3)
    at runAndTransformResultsToJestFormat (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:176:21)
    at jestAdapter (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:109:19)
    at runTestInternal (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:380:16)
    at runTest (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/runTest.js:472:34)
    at Object.worker (/var/lib/jenkins/workspace/elastic+kibana+pipeline-pull-request/kibana/node_modules/jest-runner/build/testWorker.js:133:12)

Kibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/monitoring/elasticsearch/index_detail_mb·js.Monitoring app Elasticsearch index detail mb Active Indices "before all" hook for "should have an index summary with green status index with full shard allocation"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: Monitoring app
[00:00:00]           └-> "before all" hook in "Monitoring app"
[00:13:55]           └-: Elasticsearch index detail mb
[00:13:55]             └-> "before all" hook in "Elasticsearch index detail mb"
[00:13:55]             └-: Active Indices
[00:13:55]               └-> "before all" hook for "should have an index summary with green status index with full shard allocation"
[00:13:55]               └-> "before all" hook for "should have an index summary with green status index with full shard allocation"
[00:13:55]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Loading "mappings.json"
[00:13:55]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Loading "data.json.gz"
[00:13:55]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0] creating index, cause [api], templates [], shards [1]/[1]
[00:13:56]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index "metricbeat-8.0.0"
[00:13:56]                 │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] "metricbeat-8.0.0" settings {"index":{"codec":"best_compression","lifecycle":{"name":"metricbeat","rollover_alias":"metricbeat-8.0.0"},"mapping":{"total_fields":{"limit":"10000"}},"max_docvalue_fields_search":"200","number_of_replicas":"1","number_of_shards":"1","query":{"default_field":["message","tags","agent.ephemeral_id","agent.id","agent.name","agent.type","agent.version","as.organization.name","client.address","client.as.organization.name","client.domain","client.geo.city_name","client.geo.continent_name","client.geo.country_iso_code","client.geo.country_name","client.geo.name","client.geo.region_iso_code","client.geo.region_name","client.mac","client.registered_domain","client.top_level_domain","client.user.domain","client.user.email","client.user.full_name","client.user.group.domain","client.user.group.id","client.user.group.name","client.user.hash","client.user.id","client.user.name","cloud.account.id","cloud.availability_zone","cloud.instance.id","cloud.instance.name","cloud.machine.type","cloud.provider","cloud.region","container.id","container.image.name","container.image.tag","container.name","container.runtime","destination.address","destination.as.organization.name","destination.domain","destination.geo.city_name","destination.geo.continent_name","destination.geo.country_iso_code","destination.geo.country_name","destination.geo.name","destination.geo.region_iso_code","destination.geo.region_name","destination.mac","destination.registered_domain","destination.top_level_domain","destination.user.domain","destination.user.email","destination.user.full_name","destination.user.group.domain","destination.user.group.id","destination.user.group.name","destination.user.hash","destination.user.id","destination.user.name","dns.answers.class","dns.answers.data","dns.answers.name","dns.answers.type","dns.header_flags","dns.id","dns.op_code","dns.question.class","dns.question.name","dns.question.registered_domain","dns.question.subdomain","dns.question.top_level_domain","dns.question.type","dns.response_code","dns.type","ecs.version","error.code","error.id","error.message","error.stack_trace","error.type","event.action","event.category","event.code","event.dataset","event.hash","event.id","event.kind","event.module","event.original","event.outcome","event.provider","event.timezone","event.type","file.device","file.directory","file.extension","file.gid","file.group","file.hash.md5","file.hash.sha1","file.hash.sha256","file.hash.sha512","file.inode","file.mode","file.name","file.owner","file.path","file.target_path","file.type","file.uid","geo.city_name","geo.continent_name","geo.country_iso_code","geo.country_name","geo.name","geo.region_iso_code","geo.region_name","group.domain","group.id","group.name","hash.md5","hash.sha1","hash.sha256","hash.sha512","host.architecture","host.geo.city_name","host.geo.continent_name","host.geo.country_iso_code","host.geo.country_name","host.geo.name","host.geo.region_iso_code","host.geo.region_name","host.hostname","host.id","host.mac","host.name","host.os.family","host.os.full","host.os.kernel","host.os.name","host.os.platform","host.os.version","host.type","host.user.domain","host.user.email","host.user.full_name","host.user.group.domain","host.user.group.id","host.user.group.name","host.user.hash","host.user.id","host.user.name","http.request.body.content","http.request.method","http.request.referrer","http.response.body.content","http.version","log.level","log.logger","log.origin.file.name","log.origin.function","log.original","log.syslog.facility.name","log.syslog.severity.name","network.application","network.community_id","network.direction","network.iana_number","network.name","network.protocol","network.transport","network.type","observer.geo.city_name","observer.geo.continent_name","observer.geo.country_iso_code","observer.geo.country_name","observer.geo.name","observer.geo.region_iso_code","observer.geo.region_name","observer.hostname","observer.mac","observer.name","observer.os.family","observer.os.full","observer.os.kernel","observer.os.name","observer.os.platform","observer.os.version","observer.product","observer.serial_number","observer.type","observer.vendor","observer.version","organization.id","organization.name","os.family","os.full","os.kernel","os.name","os.platform","os.version","package.architecture","package.checksum","package.description","package.install_scope","package.license","package.name","package.path","package.version","process.args","text","process.executable","process.hash.md5","process.hash.sha1","process.hash.sha256","process.hash.sha512","process.name","text","text","text","text","text","process.thread.name","process.title","process.working_directory","server.address","server.as.organization.name","server.domain","server.geo.city_name","server.geo.continent_name","server.geo.country_iso_code","server.geo.country_name","server.geo.name","server.geo.region_iso_code","server.geo.region_name","server.mac","server.registered_domain","server.top_level_domain","server.user.domain","server.user.email","server.user.full_name","server.user.group.domain","server.user.group.id","server.user.group.name","server.user.hash","server.user.id","server.user.name","service.ephemeral_id","service.id","service.name","service.node.name","service.state","service.type","service.version","source.address","source.as.organization.name","source.domain","source.geo.city_name","source.geo.continent_name","source.geo.country_iso_code","source.geo.country_name","source.geo.name","source.geo.region_iso_code","source.geo.region_name","source.mac","source.registered_domain","source.top_level_domain","source.user.domain","source.user.email","source.user.full_name","source.user.group.domain","source.user.group.id","source.user.group.name","source.user.hash","source.user.id","source.user.name","threat.framework","threat.tactic.id","threat.tactic.name","threat.tactic.reference","threat.technique.id","threat.technique.name","threat.technique.reference","tracing.trace.id","tracing.transaction.id","url.domain","url.extension","url.fragment","url.full","url.original","url.password","url.path","url.query","url.registered_domain","url.scheme","url.top_level_domain","url.username","user.domain","user.email","user.full_name","user.group.domain","user.group.id","user.group.name","user.hash","user.id","user.name","user_agent.device.name","user_agent.name","text","user_agent.original","user_agent.os.family","user_agent.os.full","user_agent.os.kernel","user_agent.os.name","user_agent.os.platform","user_agent.os.version","user_agent.version","text","agent.hostname","timeseries.instance","cloud.project.id","cloud.image.id","host.os.build","host.os.codename","kubernetes.pod.name","kubernetes.pod.uid","kubernetes.namespace","kubernetes.node.name","kubernetes.replicaset.name","kubernetes.deployment.name","kubernetes.statefulset.name","kubernetes.container.name","kubernetes.container.image","jolokia.agent.version","jolokia.agent.id","jolokia.server.product","jolokia.server.version","jolokia.server.vendor","jolokia.url","metricset.name","service.address","service.hostname","type","systemd.fragment_path","systemd.unit","aerospike.namespace.name","aerospike.namespace.node.host","aerospike.namespace.node.name","apache.status.hostname","beat.id","beat.type","beat.state.service.id","beat.state.service.name","beat.state.service.version","beat.state.input.names","beat.state.beat.host","beat.state.beat.name","beat.state.beat.type","beat.state.beat.uuid","beat.state.beat.version","beat.state.cluster.uuid","beat.state.host.containerized","beat.state.host.os.kernel","beat.state.host.os.name","beat.state.host.os.platform","beat.state.host.os.version","beat.state.module.names","beat.state.output.name","beat.state.queue.name","beat.stats.beat.name","beat.stats.beat.host","beat.stats.beat.type","beat.stats.beat.uuid","beat.stats.beat.version","beat.stats.info.ephemeral_id","beat.stats.libbeat.output.type","ceph.cluster_health.overall_status","ceph.cluster_health.timechecks.round.status","ceph.mgr_osd_pool_stats.pool_name","ceph.monitor_health.health","ceph.monitor_health.name","ceph.osd_df.name","ceph.osd_df.device_class","ceph.osd_tree.name","ceph.osd_tree.type","ceph.osd_tree.children","ceph.osd_tree.status","ceph.osd_tree.device_class","ceph.osd_tree.father","ceph.pool_disk.name","couchbase.bucket.name","couchbase.bucket.type","couchbase.node.hostname","docker.container.command","docker.container.status","docker.container.tags","docker.event.status","docker.event.id","docker.event.from","docker.event.type","docker.event.action","docker.event.actor.id","docker.healthcheck.status","docker.healthcheck.event.output","docker.image.id.current","docker.image.id.parent","docker.image.tags","docker.info.id","docker.network.interface","elasticsearch.cluster.name","elasticsearch.cluster.id","elasticsearch.cluster.state.id","elasticsearch.node.id","elasticsearch.node.name","elasticsearch.ccr.remote_cluster","elasticsearch.ccr.leader.index","elasticsearch.ccr.follower.index","elasticsearch.cluster.stats.version","elasticsearch.cluster.stats.state.nodes_hash","elasticsearch.cluster.stats.state.master_node","elasticsearch.cluster.stats.state.version","elasticsearch.cluster.stats.state.state_uuid","elasticsearch.cluster.stats.status","elasticsearch.cluster.stats.license.status","elasticsearch.cluster.stats.license.type","elasticsearch.enrich.executing_policy.name","elasticsearch.enrich.executing_policy.task.task","elasticsearch.enrich.executing_policy.task.action","elasticsearch.enrich.executing_policy.task.parent_task_id","elasticsearch.index.uuid","elasticsearch.index.status","elasticsearch.index.name","elasticsearch.index.recovery.index.files.percent","elasticsearch.index.recovery.name","elasticsearch.index.recovery.type","elasticsearch.index.recovery.stage","elasticsearch.index.recovery.translog.percent","elasticsearch.index.recovery.target.transport_address","elasticsearch.index.recovery.target.id","elasticsearch.index.recovery.target.host","elasticsearch.index.recovery.target.name","elasticsearch.index.recovery.source.transport_address","elasticsearch.index.recovery.source.id","elasticsearch.index.recovery.source.host","elasticsearch.index.recovery.source.name","elasticsearch.ml.job.id","elasticsearch.ml.job.state","elasticsearch.ml.job.model_size.memory_status","elasticsearch.node.version","elasticsearch.node.jvm.version","elasticsearch.node.stats.os.cgroup.memory.control_group","elasticsearch.cluster.pending_task.source","elasticsearch.shard.state","elasticsearch.shard.relocating_node.name","elasticsearch.shard.relocating_node.id","elasticsearch.shard.source_node.name","elasticsearch.shard.source_node.uuid","etcd.api_version","etcd.leader.leader","etcd.self.id","etcd.self.leaderinfo.leader","etcd.self.leaderinfo.starttime","etcd.self.leaderinfo.uptime","etcd.self.name","etcd.self.starttime","etcd.self.state","golang.expvar.cmdline","golang.heap.cmdline","graphite.server.example","haproxy.stat.status","haproxy.stat.service_name","haproxy.stat.cookie","haproxy.stat.load_balancing_algorithm","haproxy.stat.check.status","haproxy.stat.check.health.last","haproxy.stat.proxy.name","haproxy.stat.proxy.mode","haproxy.stat.agent.status","haproxy.stat.agent.description","haproxy.stat.agent.check.description","haproxy.stat.source.address","http.response.code","http.response.phrase","kafka.broker.address","kafka.topic.name","kafka.partition.topic_id","kafka.partition.topic_broker_id","kafka.broker.mbean","kafka.consumer.mbean","kafka.consumergroup.broker.address","kafka.consumergroup.id","kafka.consumergroup.topic","kafka.consumergroup.meta","kafka.consumergroup.client.id","kafka.consumergroup.client.host","kafka.consumergroup.client.member_id","kafka.partition.topic.name","kafka.partition.broker.address","kafka.producer.mbean","kibana.settings.uuid","kibana.settings.name","kibana.settings.index","kibana.settings.host","kibana.settings.transport_address","kibana.settings.version","kibana.settings.status","kibana.settings.locale","kibana.stats.kibana.status","kibana.stats.usage.index","kibana.stats.name","kibana.stats.index","kibana.stats.host.name","kibana.stats.status","kibana.stats.os.distro","kibana.stats.os.distroRelease","kibana.stats.os.platform","kibana.stats.os.platformRelease","kibana.status.name","kibana.status.status.overall.state","kubernetes.apiserver.request.client","kubernetes.apiserver.request.resource","kubernetes.apiserver.request.subresource","kubernetes.apiserver.request.scope","kubernetes.apiserver.request.verb","kubernetes.apiserver.request.code","kubernetes.apiserver.request.content_type","kubernetes.apiserver.request.dry_run","kubernetes.apiserver.request.kind","kubernetes.apiserver.request.component","kubernetes.apiserver.request.group","kubernetes.apiserver.request.version","kubernetes.apiserver.request.handler","kubernetes.apiserver.request.method","kubernetes.apiserver.request.host","kubernetes.controllermanager.handler","kubernetes.controllermanager.code","kubernetes.controllermanager.method","kubernetes.controllermanager.host","kubernetes.controllermanager.name","kubernetes.controllermanager.zone","kubernetes.event.message","kubernetes.event.reason","kubernetes.event.type","kubernetes.event.source.component","kubernetes.event.source.host","kubernetes.event.metadata.generate_name","kubernetes.event.metadata.name","kubernetes.event.metadata.namespace","kubernetes.event.metadata.resource_version","kubernetes.event.metadata.uid","kubernetes.event.metadata.self_link","kubernetes.event.involved_object.api_version","kubernetes.event.involved_object.kind","kubernetes.event.involved_object.name","kubernetes.event.involved_object.resource_version","kubernetes.event.involved_object.uid","kubernetes.proxy.handler","kubernetes.proxy.code","kubernetes.proxy.method","kubernetes.proxy.host","kubernetes.scheduler.handler","kubernetes.scheduler.code","kubernetes.scheduler.method","kubernetes.scheduler.host","kubernetes.scheduler.name","kubernetes.scheduler.result","kubernetes.scheduler.operation","kubernetes.container.id","kubernetes.container.status.phase","kubernetes.container.status.reason","kubernetes.cronjob.name","kubernetes.cronjob.schedule","kubernetes.cronjob.concurrency","kubernetes.daemonset.name","kubernetes.node.status.ready","kubernetes.node.status.memory_pressure","kubernetes.node.status.disk_pressure","kubernetes.node.status.out_of_disk","kubernetes.node.status.pid_pressure","kubernetes.persistentvolume.name","kubernetes.persistentvolume.phase","kubernetes.persistentvolume.storage_class","kubernetes.persistentvolumeclaim.name","kubernetes.persistentvolumeclaim.volume_name","kubernetes.persistentvolumeclaim.phase","kubernetes.persistentvolumeclaim.access_mode","kubernetes.persistentvolumeclaim.storage_class","kubernetes.pod.status.phase","kubernetes.pod.status.ready","kubernetes.pod.status.scheduled","kubernetes.resourcequota.name","kubernetes.resourcequota.type","kubernetes.resourcequota.resource","kubernetes.service.name","kubernetes.service.cluster_ip","kubernetes.service.external_name","kubernetes.service.external_ip","kubernetes.service.load_balancer_ip","kubernetes.service.type","kubernetes.service.ingress_ip","kubernetes.service.ingress_hostname","kubernetes.storageclass.name","kubernetes.storageclass.provisioner","kubernetes.storageclass.reclaim_policy","kubernetes.storageclass.volume_binding_mode","kubernetes.system.container","kubernetes.volume.name","kvm.name","kvm.dommemstat.stat.name","kvm.dommemstat.name","kvm.status.state","logstash.node.state.pipeline.id","logstash.node.state.pipeline.hash","logstash.node.jvm.version","logstash.node.stats.logstash.uuid","logstash.node.stats.logstash.version","logstash.node.stats.pipelines.id","logstash.node.stats.pipelines.hash","logstash.node.stats.pipelines.queue.type","logstash.node.stats.pipelines.vertices.pipeline_ephemeral_id","logstash.node.stats.pipelines.vertices.id","mongodb.collstats.db","mongodb.collstats.collection","mongodb.collstats.name","mongodb.dbstats.db","mongodb.metrics.replication.executor.network_interface","mongodb.replstatus.set_name","mongodb.replstatus.members.primary.host","mongodb.replstatus.members.primary.optime","mongodb.replstatus.members.secondary.hosts","mongodb.replstatus.members.secondary.optimes","mongodb.replstatus.members.recovering.hosts","mongodb.replstatus.members.unknown.hosts","mongodb.replstatus.members.startup2.hosts","mongodb.replstatus.members.arbiter.hosts","mongodb.replstatus.members.down.hosts","mongodb.replstatus.members.rollback.hosts","mongodb.replstatus.members.unhealthy.hosts","mongodb.status.storage_engine.name","munin.plugin.name","mysql.galera_status.cluster.status","mysql.galera_status.connected","mysql.galera_status.evs.evict","mysql.galera_status.evs.state","mysql.galera_status.local.state","mysql.galera_status.ready","mysql.performance.events_statements.digest","mysql.performance.table_io_waits.object.schema","mysql.performance.table_io_waits.object.name","mysql.performance.table_io_waits.index.name","nats.server.id","nats.connection.name","nats.route.remote_id","nginx.stubstatus.hostname","php_fpm.pool.name","php_fpm.pool.process_manager","php_fpm.process.state","php_fpm.process.script","postgresql.activity.database.name","postgresql.activity.user.name","postgresql.activity.application_name","postgresql.activity.client.address","postgresql.activity.client.hostname","postgresql.activity.backend_type","postgresql.activity.state","postgresql.activity.query","postgresql.activity.wait_event","postgresql.activity.wait_event_type","postgresql.database.name","postgresql.statement.query.text","rabbitmq.vhost","rabbitmq.connection.name","rabbitmq.connection.state","rabbitmq.connection.type","rabbitmq.connection.host","rabbitmq.connection.peer.host","rabbitmq.connection.client_provided.name","rabbitmq.exchange.name","rabbitmq.node.name","rabbitmq.node.type","rabbitmq.queue.name","rabbitmq.queue.state","redis.info.memory.max.policy","redis.info.memory.allocator","redis.info.persistence.rdb.bgsave.last_status","redis.info.persistence.aof.bgrewrite.last_status","redis.info.persistence.aof.write.last_status","redis.info.replication.role","redis.info.replication.master.link_status","redis.info.server.git_sha1","redis.info.server.git_dirty","redis.info.server.build_id","redis.info.server.mode","redis.info.server.arch_bits","redis.info.server.multiplexing_api","redis.info.server.gcc_version","redis.info.server.run_id","redis.info.server.config_file","redis.key.name","redis.key.id","redis.key.type","redis.keyspace.id","process.state","system.diskio.name","system.diskio.serial_number","system.filesystem.device_name","system.filesystem.type","system.filesystem.mount_point","system.network.name","system.process.state","system.process.cmdline","system.process.cgroup.id","system.process.cgroup.path","system.process.cgroup.cpu.id","system.process.cgroup.cpu.path","system.process.cgroup.cpuacct.id","system.process.cgroup.cpuacct.path","system.process.cgroup.memory.id","system.process.cgroup.memory.path","system.process.cgroup.blkio.id","system.process.cgroup.blkio.path","system.raid.name","system.raid.status","system.raid.level","system.raid.sync_action","system.service.name","system.service.load_state","system.service.state","system.service.sub_state","system.service.exec_code","system.service.unit_file.state","system.service.unit_file.vendor_preset","system.socket.remote.host","system.socket.remote.etld_plus_one","system.socket.remote.host_error","system.socket.process.cmdline","system.users.id","system.users.seat","system.users.path","system.users.type","system.users.service","system.users.state","system.users.scope","system.users.remote_host","uwsgi.status.worker.status","vsphere.datastore.name","vsphere.datastore.fstype","vsphere.host.name","vsphere.host.network_names","vsphere.virtualmachine.host.id","vsphere.virtualmachine.host.hostname","vsphere.virtualmachine.name","vsphere.virtualmachine.os","vsphere.virtualmachine.network_names","windows.perfmon.instance","windows.service.id","windows.service.name","windows.service.display_name","windows.service.start_type","windows.service.start_name","windows.service.path_name","windows.service.state","windows.service.exit_code","zookeeper.mntr.hostname","zookeeper.mntr.server_state","zookeeper.server.mode","zookeeper.server.zxid","fields.*"]},"refresh_interval":"5s","routing":{"allocation":{"include":{"_tier_preference":"data_content"}}}}}
[00:13:56]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-kibana-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:13:56]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-kibana-6-2017.10.05"
[00:13:56]                 │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-kibana-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:13:56]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-alerts-6] creating index, cause [api], templates [], shards [1]/[1]
[00:13:56]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-alerts-6"
[00:13:56]                 │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-alerts-6" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:13:56]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-logstash-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:13:56]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-logstash-6-2017.10.05"
[00:13:56]                 │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-logstash-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:13:56]                 │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-es-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:13:56]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-es-6-2017.10.05"
[00:13:56]                 │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-es-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:13:56]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/MgLVNx48T7-Htr-zhajqRw] update_mapping [_doc]
[00:13:56]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/MgLVNx48T7-Htr-zhajqRw] update_mapping [_doc]
[00:13:56]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/MgLVNx48T7-Htr-zhajqRw] update_mapping [_doc]
[00:13:57]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/MgLVNx48T7-Htr-zhajqRw] update_mapping [_doc]
[00:13:57]                 │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/MgLVNx48T7-Htr-zhajqRw] update_mapping [_doc]
[00:13:58]                 │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Indexed 2930 docs into "metricbeat-8.0.0"
[00:13:58]                 │ debg replacing kibana config doc: {}
[00:13:58]                 │ debg creating user basic_monitoring_user
[00:13:58]                 │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] added user [basic_monitoring_user]
[00:13:58]                 │ debg created user basic_monitoring_user
[00:13:58]                 │ debg SecurityPage.forceLogout
[00:13:58]                 │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=100
[00:13:58]                 │ debg Already on the login page, not forcing anything
[00:13:58]                 │ debg TestSubjects.setValue(loginUsername, basic_monitoring_user)
[00:13:58]                 │ debg TestSubjects.click(loginUsername)
[00:13:58]                 │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:13:58]                 │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:13:59]                 │ debg TestSubjects.setValue(loginPassword, monitoring_user_password)
[00:13:59]                 │ debg TestSubjects.click(loginPassword)
[00:13:59]                 │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:13:59]                 │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:13:59]                 │ debg TestSubjects.click(loginSubmit)
[00:13:59]                 │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:13:59]                 │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:13:59]                 │ debg Find.waitForDeletedByCssSelector('.kibanaWelcomeLogo') with timeout=10000
[00:13:59]                 │ proc [kibana]   log   [11:55:57.759] [info][plugins][routes][security] Logging in with provider "basic" (basic)
[00:13:59]                 │ debg browser[INFO] http://localhost:61131/app/home 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:13:59]                 │
[00:13:59]                 │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:13:59]                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:14:01]                 │ debg navigating to monitoring url: http://localhost:61131/app/monitoring
[00:14:01]                 │ debg navigate to: http://localhost:61131/app/monitoring
[00:14:01]                 │ debg browser[INFO] http://localhost:61131/app/monitoring?_t=1617623759421 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:14:01]                 │
[00:14:01]                 │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:14:01]                 │ debg ... sleep(700) start
[00:14:02]                 │ debg ... sleep(700) end
[00:14:02]                 │ debg returned from get, calling refresh
[00:14:02]                 │ debg browser[INFO] http://localhost:61131/app/monitoring?_t=1617623759421#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now)) 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:14:02]                 │
[00:14:02]                 │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:14:03]                 │ debg currentUrl = http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))
[00:14:03]                 │          appUrl = http://localhost:61131/app/monitoring
[00:14:03]                 │ debg TestSubjects.find(kibanaChrome)
[00:14:03]                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:14:03]                 │ debg browser[WARNING] http://localhost:61131/41829/bundles/plugin/monitoring/monitoring.chunk.1.js 29:205493 "\"#noDataReact\" element has not been added to the DOM yet"
[00:14:03]                 │ debg ... sleep(501) start
[00:14:03]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:03]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:03]                 │ debg ... sleep(501) end
[00:14:03]                 │ debg in navigateTo url = http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))
[00:14:03]                 │ debg TestSubjects.exists(statusPageContainer)
[00:14:03]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:14:06]                 │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:14:06]                 │ debg pauseAutoRefresh
[00:14:06]                 │ debg setting menu open state [name=QuickSelectTime Menu] [state=open]
[00:14:06]                 │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:14:06]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=1000
[00:14:08]                 │ debg --- retry.tryForTime error: [data-test-subj="superDatePickerQuickMenu"] is not displayed
[00:14:08]                 │ debg TestSubjects.click(superDatePickerToggleQuickMenuButton)
[00:14:08]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:14:08]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:14:08]                 │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:14:08]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=10000
[00:14:08]                 │ debg TestSubjects.getAttribute(superDatePickerRefreshIntervalInput, value, tryTimeout=120000, findTimeout=10000)
[00:14:08]                 │ debg TestSubjects.find(superDatePickerRefreshIntervalInput)
[00:14:08]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerRefreshIntervalInput"]') with timeout=10000
[00:14:08]                 │ debg TestSubjects.find(superDatePickerRefreshIntervalUnitsSelect)
[00:14:08]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerRefreshIntervalUnitsSelect"]') with timeout=10000
[00:14:08]                 │ debg Find.allDescendantDisplayedByCssSelector('option')
[00:14:08]                 │ debg TestSubjects.getVisibleText(superDatePickerToggleRefreshButton)
[00:14:08]                 │ debg TestSubjects.find(superDatePickerToggleRefreshButton)
[00:14:08]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:14:08]                 │ debg pause auto refresh
[00:14:08]                 │ debg TestSubjects.click(superDatePickerToggleRefreshButton)
[00:14:08]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:14:08]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:14:09]                 │ debg setting menu open state [name=QuickSelectTime Menu] [state=closed]
[00:14:09]                 │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:14:09]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=1000
[00:14:09]                 │ debg TestSubjects.click(superDatePickerToggleQuickMenuButton)
[00:14:09]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:14:09]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:14:09]                 │ debg Find.waitForDeletedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=10000
[00:14:10]                 │ debg Setting absolute range to Oct 5, 2017 @ 20:31:48.354 to Oct 5, 2017 @ 20:35:12.176
[00:14:10]                 │ debg TestSubjects.exists(superDatePickerToggleQuickMenuButton)
[00:14:10]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=20000
[00:14:10]                 │ debg TestSubjects.exists(superDatePickerShowDatesButton)
[00:14:10]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=2500
[00:14:10]                 │ debg TestSubjects.click(superDatePickerShowDatesButton)
[00:14:10]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=10000
[00:14:10]                 │ debg TestSubjects.exists(superDatePickerstartDatePopoverButton)
[00:14:10]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=2500
[00:14:10]                 │ debg TestSubjects.click(superDatePickerendDatePopoverButton)
[00:14:10]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerendDatePopoverButton"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerendDatePopoverButton"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('div.euiPopover__panel-isOpen') with timeout=10000
[00:14:10]                 │ debg TestSubjects.click(superDatePickerAbsoluteTab)
[00:14:10]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:14:10]                 │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:14:10]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:10]                 │ debg TestSubjects.setValue(superDatePickerAbsoluteDateInput, Oct 5, 2017 @ 20:35:12.176)
[00:14:10]                 │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:14:10]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:10]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:11]                 │ debg TestSubjects.click(superDatePickerstartDatePopoverButton)
[00:14:11]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=10000
[00:14:11]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=10000
[00:14:11]                 │ debg Find.waitForElementStale with timeout=10000
[00:14:11]                 │ debg Find.findByCssSelector('div.euiPopover__panel-isOpen') with timeout=10000
[00:14:11]                 │ debg TestSubjects.click(superDatePickerAbsoluteTab)
[00:14:11]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:14:11]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:14:11]                 │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:14:11]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:11]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:11]                 │ debg TestSubjects.setValue(superDatePickerAbsoluteDateInput, Oct 5, 2017 @ 20:31:48.354)
[00:14:11]                 │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:14:11]                 │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:11]                 │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:14:11]                 │ debg TestSubjects.exists(superDatePickerApplyTimeButton)
[00:14:11]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=2500
[00:14:14]                 │ debg --- retry.tryForTime error: [data-test-subj="superDatePickerApplyTimeButton"] is not displayed
[00:14:14]                 │ debg TestSubjects.click(querySubmitButton)
[00:14:14]                 │ debg Find.clickByCssSelector('[data-test-subj="querySubmitButton"]') with timeout=10000
[00:14:14]                 │ debg Find.findByCssSelector('[data-test-subj="querySubmitButton"]') with timeout=10000
[00:14:15]                 │ debg Find.waitForElementStale with timeout=10000
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:14:15]                 │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:14:15]                 │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:14:15]                 │ debg TestSubjects.click(clusterItemContainerElasticsearch > esNumberOfIndices)
[00:14:15]                 │ debg Find.clickByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:14:15]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:14:25]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:14:25]                 │      Wait timed out after 10007ms
[00:14:25]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:14:35]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:14:35]                 │      Wait timed out after 10006ms
[00:14:36]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:14:46]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:14:46]                 │      Wait timed out after 10008ms
[00:14:46]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:14:56]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:14:56]                 │      Wait timed out after 10032ms
[00:14:57]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:15:07]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:15:07]                 │      Wait timed out after 10007ms
[00:15:07]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:15:17]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:15:17]                 │      Wait timed out after 10032ms
[00:15:18]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:15:28]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:15:28]                 │      Wait timed out after 10014ms
[00:15:28]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:15:39]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:15:39]                 │      Wait timed out after 10058ms
[00:15:39]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:15:49]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:15:49]                 │      Wait timed out after 10023ms
[00:15:50]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:16:00]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:16:00]                 │      Wait timed out after 10049ms
[00:16:00]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:16:10]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:16:10]                 │      Wait timed out after 10023ms
[00:16:11]                 │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"]') with timeout=10000
[00:16:21]                 │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:16:21]                 │      Wait timed out after 10022ms
[00:16:21]                 │ info Taking screenshot "/dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/screenshots/failure/Monitoring app Elasticsearch index detail mb Active Indices _before all_ hook for _should have an index summary with green status index with full shard allocation_.png"
[00:16:21]                 │ info Current URL is: http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!t,value:10000),time:(from:%272017-10-05T20:31:48.354Z%27,to:%272017-10-05T20:35:12.176Z%27))
[00:16:21]                 │ info Saving page source to: /dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/failure_debug/html/Monitoring app Elasticsearch index detail mb Active Indices _before all_ hook for _should have an index summary with green status index with full shard allocation_.html
[00:16:21]                 └- ✖ fail: Monitoring app Elasticsearch index detail mb Active Indices "before all" hook for "should have an index summary with green status index with full shard allocation"
[00:16:21]                 │      Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
[00:16:21]                 │ Wait timed out after 10022ms
[00:16:21]                 │     at /dev/shm/workspace/parallel/13/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
[00:16:21]                 │     at runMicrotasks (<anonymous>)
[00:16:21]                 │     at processTicksAndRejections (internal/process/task_queues.js:93:5)
[00:16:21]                 │       at onFailure (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:17:9)
[00:16:21]                 │       at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:57:13)
[00:16:21]                 │       at Retry.try (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:32:14)
[00:16:21]                 │       at Proxy.clickByCssSelector (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/find.ts:420:7)
[00:16:21]                 │       at TestSubjects.click (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/test_subjects.ts:109:7)
[00:16:21]                 │       at Context.<anonymous> (test/functional/apps/monitoring/elasticsearch/index_detail_mb.js:33:9)
[00:16:21]                 │       at Object.apply (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
[00:16:21]                 │ 
[00:16:21]                 │ 

Stack Trace

Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esNumberOfIndices"])
Wait timed out after 10022ms
    at /dev/shm/workspace/parallel/13/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at onFailure (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:17:9)
    at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:57:13)
    at Retry.try (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:32:14)
    at Proxy.clickByCssSelector (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/find.ts:420:7)
    at TestSubjects.click (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/test_subjects.ts:109:7)
    at Context.<anonymous> (test/functional/apps/monitoring/elasticsearch/index_detail_mb.js:33:9)
    at Object.apply (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)

Kibana Pipeline / general / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/monitoring/elasticsearch/overview_mb·js.Monitoring app Elasticsearch overview mb "before all" hook for "should have an Elasticsearch Cluster Summary Status with correct info"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: Monitoring app
[00:00:00]           └-> "before all" hook in "Monitoring app"
[00:05:18]           └-: Elasticsearch overview mb
[00:05:18]             └-> "before all" hook for "should have an Elasticsearch Cluster Summary Status with correct info"
[00:05:18]             └-> "before all" hook for "should have an Elasticsearch Cluster Summary Status with correct info"
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Loading "mappings.json"
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Loading "data.json.gz"
[00:05:18]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0] creating index, cause [api], templates [], shards [1]/[1]
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index "metricbeat-8.0.0"
[00:05:18]               │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] "metricbeat-8.0.0" settings {"index":{"codec":"best_compression","lifecycle":{"name":"metricbeat","rollover_alias":"metricbeat-8.0.0"},"mapping":{"total_fields":{"limit":"10000"}},"max_docvalue_fields_search":"200","number_of_replicas":"1","number_of_shards":"1","query":{"default_field":["message","tags","agent.ephemeral_id","agent.id","agent.name","agent.type","agent.version","as.organization.name","client.address","client.as.organization.name","client.domain","client.geo.city_name","client.geo.continent_name","client.geo.country_iso_code","client.geo.country_name","client.geo.name","client.geo.region_iso_code","client.geo.region_name","client.mac","client.registered_domain","client.top_level_domain","client.user.domain","client.user.email","client.user.full_name","client.user.group.domain","client.user.group.id","client.user.group.name","client.user.hash","client.user.id","client.user.name","cloud.account.id","cloud.availability_zone","cloud.instance.id","cloud.instance.name","cloud.machine.type","cloud.provider","cloud.region","container.id","container.image.name","container.image.tag","container.name","container.runtime","destination.address","destination.as.organization.name","destination.domain","destination.geo.city_name","destination.geo.continent_name","destination.geo.country_iso_code","destination.geo.country_name","destination.geo.name","destination.geo.region_iso_code","destination.geo.region_name","destination.mac","destination.registered_domain","destination.top_level_domain","destination.user.domain","destination.user.email","destination.user.full_name","destination.user.group.domain","destination.user.group.id","destination.user.group.name","destination.user.hash","destination.user.id","destination.user.name","dns.answers.class","dns.answers.data","dns.answers.name","dns.answers.type","dns.header_flags","dns.id","dns.op_code","dns.question.class","dns.question.name","dns.question.registered_domain","dns.question.subdomain","dns.question.top_level_domain","dns.question.type","dns.response_code","dns.type","ecs.version","error.code","error.id","error.message","error.stack_trace","error.type","event.action","event.category","event.code","event.dataset","event.hash","event.id","event.kind","event.module","event.original","event.outcome","event.provider","event.timezone","event.type","file.device","file.directory","file.extension","file.gid","file.group","file.hash.md5","file.hash.sha1","file.hash.sha256","file.hash.sha512","file.inode","file.mode","file.name","file.owner","file.path","file.target_path","file.type","file.uid","geo.city_name","geo.continent_name","geo.country_iso_code","geo.country_name","geo.name","geo.region_iso_code","geo.region_name","group.domain","group.id","group.name","hash.md5","hash.sha1","hash.sha256","hash.sha512","host.architecture","host.geo.city_name","host.geo.continent_name","host.geo.country_iso_code","host.geo.country_name","host.geo.name","host.geo.region_iso_code","host.geo.region_name","host.hostname","host.id","host.mac","host.name","host.os.family","host.os.full","host.os.kernel","host.os.name","host.os.platform","host.os.version","host.type","host.user.domain","host.user.email","host.user.full_name","host.user.group.domain","host.user.group.id","host.user.group.name","host.user.hash","host.user.id","host.user.name","http.request.body.content","http.request.method","http.request.referrer","http.response.body.content","http.version","log.level","log.logger","log.origin.file.name","log.origin.function","log.original","log.syslog.facility.name","log.syslog.severity.name","network.application","network.community_id","network.direction","network.iana_number","network.name","network.protocol","network.transport","network.type","observer.geo.city_name","observer.geo.continent_name","observer.geo.country_iso_code","observer.geo.country_name","observer.geo.name","observer.geo.region_iso_code","observer.geo.region_name","observer.hostname","observer.mac","observer.name","observer.os.family","observer.os.full","observer.os.kernel","observer.os.name","observer.os.platform","observer.os.version","observer.product","observer.serial_number","observer.type","observer.vendor","observer.version","organization.id","organization.name","os.family","os.full","os.kernel","os.name","os.platform","os.version","package.architecture","package.checksum","package.description","package.install_scope","package.license","package.name","package.path","package.version","process.args","text","process.executable","process.hash.md5","process.hash.sha1","process.hash.sha256","process.hash.sha512","process.name","text","text","text","text","text","process.thread.name","process.title","process.working_directory","server.address","server.as.organization.name","server.domain","server.geo.city_name","server.geo.continent_name","server.geo.country_iso_code","server.geo.country_name","server.geo.name","server.geo.region_iso_code","server.geo.region_name","server.mac","server.registered_domain","server.top_level_domain","server.user.domain","server.user.email","server.user.full_name","server.user.group.domain","server.user.group.id","server.user.group.name","server.user.hash","server.user.id","server.user.name","service.ephemeral_id","service.id","service.name","service.node.name","service.state","service.type","service.version","source.address","source.as.organization.name","source.domain","source.geo.city_name","source.geo.continent_name","source.geo.country_iso_code","source.geo.country_name","source.geo.name","source.geo.region_iso_code","source.geo.region_name","source.mac","source.registered_domain","source.top_level_domain","source.user.domain","source.user.email","source.user.full_name","source.user.group.domain","source.user.group.id","source.user.group.name","source.user.hash","source.user.id","source.user.name","threat.framework","threat.tactic.id","threat.tactic.name","threat.tactic.reference","threat.technique.id","threat.technique.name","threat.technique.reference","tracing.trace.id","tracing.transaction.id","url.domain","url.extension","url.fragment","url.full","url.original","url.password","url.path","url.query","url.registered_domain","url.scheme","url.top_level_domain","url.username","user.domain","user.email","user.full_name","user.group.domain","user.group.id","user.group.name","user.hash","user.id","user.name","user_agent.device.name","user_agent.name","text","user_agent.original","user_agent.os.family","user_agent.os.full","user_agent.os.kernel","user_agent.os.name","user_agent.os.platform","user_agent.os.version","user_agent.version","text","agent.hostname","timeseries.instance","cloud.project.id","cloud.image.id","host.os.build","host.os.codename","kubernetes.pod.name","kubernetes.pod.uid","kubernetes.namespace","kubernetes.node.name","kubernetes.replicaset.name","kubernetes.deployment.name","kubernetes.statefulset.name","kubernetes.container.name","kubernetes.container.image","jolokia.agent.version","jolokia.agent.id","jolokia.server.product","jolokia.server.version","jolokia.server.vendor","jolokia.url","metricset.name","service.address","service.hostname","type","systemd.fragment_path","systemd.unit","aerospike.namespace.name","aerospike.namespace.node.host","aerospike.namespace.node.name","apache.status.hostname","beat.id","beat.type","beat.state.service.id","beat.state.service.name","beat.state.service.version","beat.state.input.names","beat.state.beat.host","beat.state.beat.name","beat.state.beat.type","beat.state.beat.uuid","beat.state.beat.version","beat.state.cluster.uuid","beat.state.host.containerized","beat.state.host.os.kernel","beat.state.host.os.name","beat.state.host.os.platform","beat.state.host.os.version","beat.state.module.names","beat.state.output.name","beat.state.queue.name","beat.stats.beat.name","beat.stats.beat.host","beat.stats.beat.type","beat.stats.beat.uuid","beat.stats.beat.version","beat.stats.info.ephemeral_id","beat.stats.libbeat.output.type","ceph.cluster_health.overall_status","ceph.cluster_health.timechecks.round.status","ceph.mgr_osd_pool_stats.pool_name","ceph.monitor_health.health","ceph.monitor_health.name","ceph.osd_df.name","ceph.osd_df.device_class","ceph.osd_tree.name","ceph.osd_tree.type","ceph.osd_tree.children","ceph.osd_tree.status","ceph.osd_tree.device_class","ceph.osd_tree.father","ceph.pool_disk.name","couchbase.bucket.name","couchbase.bucket.type","couchbase.node.hostname","docker.container.command","docker.container.status","docker.container.tags","docker.event.status","docker.event.id","docker.event.from","docker.event.type","docker.event.action","docker.event.actor.id","docker.healthcheck.status","docker.healthcheck.event.output","docker.image.id.current","docker.image.id.parent","docker.image.tags","docker.info.id","docker.network.interface","elasticsearch.cluster.name","elasticsearch.cluster.id","elasticsearch.cluster.state.id","elasticsearch.node.id","elasticsearch.node.name","elasticsearch.ccr.remote_cluster","elasticsearch.ccr.leader.index","elasticsearch.ccr.follower.index","elasticsearch.cluster.stats.version","elasticsearch.cluster.stats.state.nodes_hash","elasticsearch.cluster.stats.state.master_node","elasticsearch.cluster.stats.state.version","elasticsearch.cluster.stats.state.state_uuid","elasticsearch.cluster.stats.status","elasticsearch.cluster.stats.license.status","elasticsearch.cluster.stats.license.type","elasticsearch.enrich.executing_policy.name","elasticsearch.enrich.executing_policy.task.task","elasticsearch.enrich.executing_policy.task.action","elasticsearch.enrich.executing_policy.task.parent_task_id","elasticsearch.index.uuid","elasticsearch.index.status","elasticsearch.index.name","elasticsearch.index.recovery.index.files.percent","elasticsearch.index.recovery.name","elasticsearch.index.recovery.type","elasticsearch.index.recovery.stage","elasticsearch.index.recovery.translog.percent","elasticsearch.index.recovery.target.transport_address","elasticsearch.index.recovery.target.id","elasticsearch.index.recovery.target.host","elasticsearch.index.recovery.target.name","elasticsearch.index.recovery.source.transport_address","elasticsearch.index.recovery.source.id","elasticsearch.index.recovery.source.host","elasticsearch.index.recovery.source.name","elasticsearch.ml.job.id","elasticsearch.ml.job.state","elasticsearch.ml.job.model_size.memory_status","elasticsearch.node.version","elasticsearch.node.jvm.version","elasticsearch.node.stats.os.cgroup.memory.control_group","elasticsearch.cluster.pending_task.source","elasticsearch.shard.state","elasticsearch.shard.relocating_node.name","elasticsearch.shard.relocating_node.id","elasticsearch.shard.source_node.name","elasticsearch.shard.source_node.uuid","etcd.api_version","etcd.leader.leader","etcd.self.id","etcd.self.leaderinfo.leader","etcd.self.leaderinfo.starttime","etcd.self.leaderinfo.uptime","etcd.self.name","etcd.self.starttime","etcd.self.state","golang.expvar.cmdline","golang.heap.cmdline","graphite.server.example","haproxy.stat.status","haproxy.stat.service_name","haproxy.stat.cookie","haproxy.stat.load_balancing_algorithm","haproxy.stat.check.status","haproxy.stat.check.health.last","haproxy.stat.proxy.name","haproxy.stat.proxy.mode","haproxy.stat.agent.status","haproxy.stat.agent.description","haproxy.stat.agent.check.description","haproxy.stat.source.address","http.response.code","http.response.phrase","kafka.broker.address","kafka.topic.name","kafka.partition.topic_id","kafka.partition.topic_broker_id","kafka.broker.mbean","kafka.consumer.mbean","kafka.consumergroup.broker.address","kafka.consumergroup.id","kafka.consumergroup.topic","kafka.consumergroup.meta","kafka.consumergroup.client.id","kafka.consumergroup.client.host","kafka.consumergroup.client.member_id","kafka.partition.topic.name","kafka.partition.broker.address","kafka.producer.mbean","kibana.settings.uuid","kibana.settings.name","kibana.settings.index","kibana.settings.host","kibana.settings.transport_address","kibana.settings.version","kibana.settings.status","kibana.settings.locale","kibana.stats.kibana.status","kibana.stats.usage.index","kibana.stats.name","kibana.stats.index","kibana.stats.host.name","kibana.stats.status","kibana.stats.os.distro","kibana.stats.os.distroRelease","kibana.stats.os.platform","kibana.stats.os.platformRelease","kibana.status.name","kibana.status.status.overall.state","kubernetes.apiserver.request.client","kubernetes.apiserver.request.resource","kubernetes.apiserver.request.subresource","kubernetes.apiserver.request.scope","kubernetes.apiserver.request.verb","kubernetes.apiserver.request.code","kubernetes.apiserver.request.content_type","kubernetes.apiserver.request.dry_run","kubernetes.apiserver.request.kind","kubernetes.apiserver.request.component","kubernetes.apiserver.request.group","kubernetes.apiserver.request.version","kubernetes.apiserver.request.handler","kubernetes.apiserver.request.method","kubernetes.apiserver.request.host","kubernetes.controllermanager.handler","kubernetes.controllermanager.code","kubernetes.controllermanager.method","kubernetes.controllermanager.host","kubernetes.controllermanager.name","kubernetes.controllermanager.zone","kubernetes.event.message","kubernetes.event.reason","kubernetes.event.type","kubernetes.event.source.component","kubernetes.event.source.host","kubernetes.event.metadata.generate_name","kubernetes.event.metadata.name","kubernetes.event.metadata.namespace","kubernetes.event.metadata.resource_version","kubernetes.event.metadata.uid","kubernetes.event.metadata.self_link","kubernetes.event.involved_object.api_version","kubernetes.event.involved_object.kind","kubernetes.event.involved_object.name","kubernetes.event.involved_object.resource_version","kubernetes.event.involved_object.uid","kubernetes.proxy.handler","kubernetes.proxy.code","kubernetes.proxy.method","kubernetes.proxy.host","kubernetes.scheduler.handler","kubernetes.scheduler.code","kubernetes.scheduler.method","kubernetes.scheduler.host","kubernetes.scheduler.name","kubernetes.scheduler.result","kubernetes.scheduler.operation","kubernetes.container.id","kubernetes.container.status.phase","kubernetes.container.status.reason","kubernetes.cronjob.name","kubernetes.cronjob.schedule","kubernetes.cronjob.concurrency","kubernetes.daemonset.name","kubernetes.node.status.ready","kubernetes.node.status.memory_pressure","kubernetes.node.status.disk_pressure","kubernetes.node.status.out_of_disk","kubernetes.node.status.pid_pressure","kubernetes.persistentvolume.name","kubernetes.persistentvolume.phase","kubernetes.persistentvolume.storage_class","kubernetes.persistentvolumeclaim.name","kubernetes.persistentvolumeclaim.volume_name","kubernetes.persistentvolumeclaim.phase","kubernetes.persistentvolumeclaim.access_mode","kubernetes.persistentvolumeclaim.storage_class","kubernetes.pod.status.phase","kubernetes.pod.status.ready","kubernetes.pod.status.scheduled","kubernetes.resourcequota.name","kubernetes.resourcequota.type","kubernetes.resourcequota.resource","kubernetes.service.name","kubernetes.service.cluster_ip","kubernetes.service.external_name","kubernetes.service.external_ip","kubernetes.service.load_balancer_ip","kubernetes.service.type","kubernetes.service.ingress_ip","kubernetes.service.ingress_hostname","kubernetes.storageclass.name","kubernetes.storageclass.provisioner","kubernetes.storageclass.reclaim_policy","kubernetes.storageclass.volume_binding_mode","kubernetes.system.container","kubernetes.volume.name","kvm.name","kvm.dommemstat.stat.name","kvm.dommemstat.name","kvm.status.state","logstash.node.state.pipeline.id","logstash.node.state.pipeline.hash","logstash.node.jvm.version","logstash.node.stats.logstash.uuid","logstash.node.stats.logstash.version","logstash.node.stats.pipelines.id","logstash.node.stats.pipelines.hash","logstash.node.stats.pipelines.queue.type","logstash.node.stats.pipelines.vertices.pipeline_ephemeral_id","logstash.node.stats.pipelines.vertices.id","mongodb.collstats.db","mongodb.collstats.collection","mongodb.collstats.name","mongodb.dbstats.db","mongodb.metrics.replication.executor.network_interface","mongodb.replstatus.set_name","mongodb.replstatus.members.primary.host","mongodb.replstatus.members.primary.optime","mongodb.replstatus.members.secondary.hosts","mongodb.replstatus.members.secondary.optimes","mongodb.replstatus.members.recovering.hosts","mongodb.replstatus.members.unknown.hosts","mongodb.replstatus.members.startup2.hosts","mongodb.replstatus.members.arbiter.hosts","mongodb.replstatus.members.down.hosts","mongodb.replstatus.members.rollback.hosts","mongodb.replstatus.members.unhealthy.hosts","mongodb.status.storage_engine.name","munin.plugin.name","mysql.galera_status.cluster.status","mysql.galera_status.connected","mysql.galera_status.evs.evict","mysql.galera_status.evs.state","mysql.galera_status.local.state","mysql.galera_status.ready","mysql.performance.events_statements.digest","mysql.performance.table_io_waits.object.schema","mysql.performance.table_io_waits.object.name","mysql.performance.table_io_waits.index.name","nats.server.id","nats.connection.name","nats.route.remote_id","nginx.stubstatus.hostname","php_fpm.pool.name","php_fpm.pool.process_manager","php_fpm.process.state","php_fpm.process.script","postgresql.activity.database.name","postgresql.activity.user.name","postgresql.activity.application_name","postgresql.activity.client.address","postgresql.activity.client.hostname","postgresql.activity.backend_type","postgresql.activity.state","postgresql.activity.query","postgresql.activity.wait_event","postgresql.activity.wait_event_type","postgresql.database.name","postgresql.statement.query.text","rabbitmq.vhost","rabbitmq.connection.name","rabbitmq.connection.state","rabbitmq.connection.type","rabbitmq.connection.host","rabbitmq.connection.peer.host","rabbitmq.connection.client_provided.name","rabbitmq.exchange.name","rabbitmq.node.name","rabbitmq.node.type","rabbitmq.queue.name","rabbitmq.queue.state","redis.info.memory.max.policy","redis.info.memory.allocator","redis.info.persistence.rdb.bgsave.last_status","redis.info.persistence.aof.bgrewrite.last_status","redis.info.persistence.aof.write.last_status","redis.info.replication.role","redis.info.replication.master.link_status","redis.info.server.git_sha1","redis.info.server.git_dirty","redis.info.server.build_id","redis.info.server.mode","redis.info.server.arch_bits","redis.info.server.multiplexing_api","redis.info.server.gcc_version","redis.info.server.run_id","redis.info.server.config_file","redis.key.name","redis.key.id","redis.key.type","redis.keyspace.id","process.state","system.diskio.name","system.diskio.serial_number","system.filesystem.device_name","system.filesystem.type","system.filesystem.mount_point","system.network.name","system.process.state","system.process.cmdline","system.process.cgroup.id","system.process.cgroup.path","system.process.cgroup.cpu.id","system.process.cgroup.cpu.path","system.process.cgroup.cpuacct.id","system.process.cgroup.cpuacct.path","system.process.cgroup.memory.id","system.process.cgroup.memory.path","system.process.cgroup.blkio.id","system.process.cgroup.blkio.path","system.raid.name","system.raid.status","system.raid.level","system.raid.sync_action","system.service.name","system.service.load_state","system.service.state","system.service.sub_state","system.service.exec_code","system.service.unit_file.state","system.service.unit_file.vendor_preset","system.socket.remote.host","system.socket.remote.etld_plus_one","system.socket.remote.host_error","system.socket.process.cmdline","system.users.id","system.users.seat","system.users.path","system.users.type","system.users.service","system.users.state","system.users.scope","system.users.remote_host","uwsgi.status.worker.status","vsphere.datastore.name","vsphere.datastore.fstype","vsphere.host.name","vsphere.host.network_names","vsphere.virtualmachine.host.id","vsphere.virtualmachine.host.hostname","vsphere.virtualmachine.name","vsphere.virtualmachine.os","vsphere.virtualmachine.network_names","windows.perfmon.instance","windows.service.id","windows.service.name","windows.service.display_name","windows.service.start_type","windows.service.start_name","windows.service.path_name","windows.service.state","windows.service.exit_code","zookeeper.mntr.hostname","zookeeper.mntr.server_state","zookeeper.server.mode","zookeeper.server.zxid","fields.*"]},"refresh_interval":"5s","routing":{"allocation":{"include":{"_tier_preference":"data_content"}}}}}
[00:05:18]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-kibana-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-kibana-6-2017.10.05"
[00:05:18]               │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-kibana-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:05:18]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-alerts-6] creating index, cause [api], templates [], shards [1]/[1]
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-alerts-6"
[00:05:18]               │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-alerts-6" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:05:18]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-logstash-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-logstash-6-2017.10.05"
[00:05:18]               │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-logstash-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:05:18]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [.monitoring-es-6-2017.10.05] creating index, cause [api], templates [], shards [1]/[1]
[00:05:18]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Created index ".monitoring-es-6-2017.10.05"
[00:05:18]               │ debg [monitoring/singlecluster-three-nodes-shard-relocation_mb] ".monitoring-es-6-2017.10.05" settings {"index":{"codec":"best_compression","format":"6","number_of_replicas":"1","number_of_shards":"1"}}
[00:05:18]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/gR9eWDhARWKmVcAWUjiBdg] update_mapping [_doc]
[00:05:19]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/gR9eWDhARWKmVcAWUjiBdg] update_mapping [_doc]
[00:05:19]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/gR9eWDhARWKmVcAWUjiBdg] update_mapping [_doc]
[00:05:19]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/gR9eWDhARWKmVcAWUjiBdg] update_mapping [_doc]
[00:05:19]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] [metricbeat-8.0.0/gR9eWDhARWKmVcAWUjiBdg] update_mapping [_doc]
[00:05:20]               │ info [monitoring/singlecluster-three-nodes-shard-relocation_mb] Indexed 2930 docs into "metricbeat-8.0.0"
[00:05:20]               │ debg replacing kibana config doc: {}
[00:05:20]               │ debg creating user basic_monitoring_user
[00:05:20]               │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-ubuntu-16-tests-xxl-1617619916248654439] added user [basic_monitoring_user]
[00:05:20]               │ debg created user basic_monitoring_user
[00:05:20]               │ debg SecurityPage.forceLogout
[00:05:20]               │ debg Find.existsByDisplayedByCssSelector('.login-form') with timeout=100
[00:05:20]               │ debg Already on the login page, not forcing anything
[00:05:20]               │ debg TestSubjects.setValue(loginUsername, basic_monitoring_user)
[00:05:20]               │ debg TestSubjects.click(loginUsername)
[00:05:20]               │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:05:20]               │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:05:21]               │ debg TestSubjects.setValue(loginPassword, monitoring_user_password)
[00:05:21]               │ debg TestSubjects.click(loginPassword)
[00:05:21]               │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:05:21]               │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:05:21]               │ debg TestSubjects.click(loginSubmit)
[00:05:21]               │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:05:21]               │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:05:21]               │ debg Find.waitForDeletedByCssSelector('.kibanaWelcomeLogo') with timeout=10000
[00:05:21]               │ proc [kibana]   log   [11:38:01.442] [info][plugins][routes][security] Logging in with provider "basic" (basic)
[00:05:22]               │ debg browser[INFO] http://localhost:61131/app/home 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:05:22]               │
[00:05:22]               │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:05:22]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:05:23]               │ debg navigating to monitoring url: http://localhost:61131/app/monitoring
[00:05:23]               │ debg navigate to: http://localhost:61131/app/monitoring
[00:05:24]               │ debg browser[INFO] http://localhost:61131/app/monitoring?_t=1617622683557 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:05:24]               │
[00:05:24]               │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:05:24]               │ debg ... sleep(700) start
[00:05:25]               │ debg ... sleep(700) end
[00:05:25]               │ debg returned from get, calling refresh
[00:05:25]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:25]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:25]               │ debg browser[INFO] http://localhost:61131/app/monitoring?_t=1617622683557#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now)) 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:05:25]               │
[00:05:25]               │ debg browser[INFO] http://localhost:61131/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:05:25]               │ debg currentUrl = http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))
[00:05:25]               │          appUrl = http://localhost:61131/app/monitoring
[00:05:25]               │ debg TestSubjects.find(kibanaChrome)
[00:05:25]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:05:26]               │ debg ... sleep(501) start
[00:05:26]               │ debg browser[WARNING] http://localhost:61131/41829/bundles/plugin/monitoring/monitoring.chunk.1.js 29:205493 "\"#noDataReact\" element has not been added to the DOM yet"
[00:05:26]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:26]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:27]               │ debg ... sleep(501) end
[00:05:27]               │ debg in navigateTo url = http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))
[00:05:27]               │ debg TestSubjects.exists(statusPageContainer)
[00:05:27]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="statusPageContainer"]') with timeout=2500
[00:05:29]               │ debg --- retry.tryForTime error: [data-test-subj="statusPageContainer"] is not displayed
[00:05:30]               │ debg pauseAutoRefresh
[00:05:30]               │ debg setting menu open state [name=QuickSelectTime Menu] [state=open]
[00:05:30]               │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:05:30]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=1000
[00:05:31]               │ debg --- retry.tryForTime error: [data-test-subj="superDatePickerQuickMenu"] is not displayed
[00:05:31]               │ debg TestSubjects.click(superDatePickerToggleQuickMenuButton)
[00:05:31]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:05:31]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:05:31]               │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:05:31]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=10000
[00:05:32]               │ debg TestSubjects.getAttribute(superDatePickerRefreshIntervalInput, value, tryTimeout=120000, findTimeout=10000)
[00:05:32]               │ debg TestSubjects.find(superDatePickerRefreshIntervalInput)
[00:05:32]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerRefreshIntervalInput"]') with timeout=10000
[00:05:32]               │ debg TestSubjects.find(superDatePickerRefreshIntervalUnitsSelect)
[00:05:32]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerRefreshIntervalUnitsSelect"]') with timeout=10000
[00:05:32]               │ debg Find.allDescendantDisplayedByCssSelector('option')
[00:05:32]               │ debg TestSubjects.getVisibleText(superDatePickerToggleRefreshButton)
[00:05:32]               │ debg TestSubjects.find(superDatePickerToggleRefreshButton)
[00:05:32]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:05:32]               │ debg pause auto refresh
[00:05:32]               │ debg TestSubjects.click(superDatePickerToggleRefreshButton)
[00:05:32]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:05:32]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleRefreshButton"]') with timeout=10000
[00:05:32]               │ debg setting menu open state [name=QuickSelectTime Menu] [state=closed]
[00:05:32]               │ debg TestSubjects.exists(superDatePickerQuickMenu)
[00:05:32]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=1000
[00:05:32]               │ debg TestSubjects.click(superDatePickerToggleQuickMenuButton)
[00:05:32]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:05:32]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=10000
[00:05:32]               │ debg Find.waitForDeletedByCssSelector('[data-test-subj="superDatePickerQuickMenu"]') with timeout=10000
[00:05:33]               │ debg Setting absolute range to Oct 5, 2017 @ 20:31:48.354 to Oct 5, 2017 @ 20:35:12.176
[00:05:33]               │ debg TestSubjects.exists(superDatePickerToggleQuickMenuButton)
[00:05:33]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerToggleQuickMenuButton"]') with timeout=20000
[00:05:33]               │ debg TestSubjects.exists(superDatePickerShowDatesButton)
[00:05:33]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=2500
[00:05:33]               │ debg TestSubjects.click(superDatePickerShowDatesButton)
[00:05:33]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerShowDatesButton"]') with timeout=10000
[00:05:33]               │ debg TestSubjects.exists(superDatePickerstartDatePopoverButton)
[00:05:33]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=2500
[00:05:33]               │ debg TestSubjects.click(superDatePickerendDatePopoverButton)
[00:05:33]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerendDatePopoverButton"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerendDatePopoverButton"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('div.euiPopover__panel-isOpen') with timeout=10000
[00:05:33]               │ debg TestSubjects.click(superDatePickerAbsoluteTab)
[00:05:33]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:05:33]               │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:05:33]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:33]               │ debg TestSubjects.setValue(superDatePickerAbsoluteDateInput, Oct 5, 2017 @ 20:35:12.176)
[00:05:33]               │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:05:33]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:33]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:34]               │ debg TestSubjects.click(superDatePickerstartDatePopoverButton)
[00:05:34]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=10000
[00:05:34]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerstartDatePopoverButton"]') with timeout=10000
[00:05:34]               │ debg Find.waitForElementStale with timeout=10000
[00:05:34]               │ debg Find.findByCssSelector('div.euiPopover__panel-isOpen') with timeout=10000
[00:05:34]               │ debg TestSubjects.click(superDatePickerAbsoluteTab)
[00:05:34]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:05:34]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteTab"]') with timeout=10000
[00:05:34]               │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:05:34]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:34]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:34]               │ debg TestSubjects.setValue(superDatePickerAbsoluteDateInput, Oct 5, 2017 @ 20:31:48.354)
[00:05:34]               │ debg TestSubjects.click(superDatePickerAbsoluteDateInput)
[00:05:34]               │ debg Find.clickByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:34]               │ debg Find.findByCssSelector('[data-test-subj="superDatePickerAbsoluteDateInput"]') with timeout=10000
[00:05:35]               │ debg TestSubjects.exists(superDatePickerApplyTimeButton)
[00:05:35]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="superDatePickerApplyTimeButton"]') with timeout=2500
[00:05:37]               │ debg --- retry.tryForTime error: [data-test-subj="superDatePickerApplyTimeButton"] is not displayed
[00:05:38]               │ debg TestSubjects.click(querySubmitButton)
[00:05:38]               │ debg Find.clickByCssSelector('[data-test-subj="querySubmitButton"]') with timeout=10000
[00:05:38]               │ debg Find.findByCssSelector('[data-test-subj="querySubmitButton"]') with timeout=10000
[00:05:38]               │ debg Find.waitForElementStale with timeout=10000
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:05:38]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/cluster - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ERROR browser[SEVERE] http://localhost:61131/api/monitoring/v1/elasticsearch_settings/check/nodes - Failed to load resource: the server responded with a status of 403 (Forbidden)
[00:05:38]               │ debg TestSubjects.click(clusterItemContainerElasticsearch > esOverview)
[00:05:38]               │ debg Find.clickByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:05:38]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:05:48]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:05:48]               │      Wait timed out after 10028ms
[00:05:49]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:05:59]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:05:59]               │      Wait timed out after 10011ms
[00:05:59]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:06:09]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:06:09]               │      Wait timed out after 10045ms
[00:06:10]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:06:20]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:06:20]               │      Wait timed out after 10044ms
[00:06:20]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:06:30]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:06:30]               │      Wait timed out after 10024ms
[00:06:31]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:06:41]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:06:41]               │      Wait timed out after 10014ms
[00:06:41]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:06:51]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:06:51]               │      Wait timed out after 10008ms
[00:06:52]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:07:02]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:02]               │      Wait timed out after 10006ms
[00:07:02]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:07:12]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:12]               │      Wait timed out after 10028ms
[00:07:13]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:07:23]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:23]               │      Wait timed out after 10006ms
[00:07:23]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:07:33]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:33]               │      Wait timed out after 10016ms
[00:07:34]               │ debg Find.findByCssSelector('[data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"]') with timeout=10000
[00:07:44]               │ debg --- retry.try error: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:44]               │      Wait timed out after 10060ms
[00:07:44]               │ info Taking screenshot "/dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/screenshots/failure/Monitoring app Elasticsearch overview mb _before all_ hook for _should have an Elasticsearch Cluster Summary Status with correct info_.png"
[00:07:45]               │ info Current URL is: http://localhost:61131/app/monitoring#/no-data?_g=(filters:!(),refreshInterval:(pause:!t,value:10000),time:(from:%272017-10-05T20:31:48.354Z%27,to:%272017-10-05T20:35:12.176Z%27))
[00:07:45]               │ info Saving page source to: /dev/shm/workspace/parallel/13/kibana/x-pack/test/functional/failure_debug/html/Monitoring app Elasticsearch overview mb _before all_ hook for _should have an Elasticsearch Cluster Summary Status with correct info_.html
[00:07:45]               └- ✖ fail: Monitoring app Elasticsearch overview mb "before all" hook for "should have an Elasticsearch Cluster Summary Status with correct info"
[00:07:45]               │      Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
[00:07:45]               │ Wait timed out after 10060ms
[00:07:45]               │     at /dev/shm/workspace/parallel/13/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
[00:07:45]               │     at runMicrotasks (<anonymous>)
[00:07:45]               │     at processTicksAndRejections (internal/process/task_queues.js:93:5)
[00:07:45]               │       at onFailure (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:17:9)
[00:07:45]               │       at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:57:13)
[00:07:45]               │       at Retry.try (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:32:14)
[00:07:45]               │       at Proxy.clickByCssSelector (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/find.ts:420:7)
[00:07:45]               │       at TestSubjects.click (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/test_subjects.ts:109:7)
[00:07:45]               │       at Context.<anonymous> (test/functional/apps/monitoring/elasticsearch/overview_mb.js:26:7)
[00:07:45]               │       at Object.apply (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
[00:07:45]               │ 
[00:07:45]               │ 

Stack Trace

Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="clusterItemContainerElasticsearch"] [data-test-subj="esOverview"])
Wait timed out after 10060ms
    at /dev/shm/workspace/parallel/13/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at onFailure (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:17:9)
    at retryForSuccess (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry_for_success.ts:57:13)
    at Retry.try (/dev/shm/workspace/parallel/13/kibana/test/common/services/retry/retry.ts:32:14)
    at Proxy.clickByCssSelector (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/find.ts:420:7)
    at TestSubjects.click (/dev/shm/workspace/parallel/13/kibana/test/functional/services/common/test_subjects.ts:109:7)
    at Context.<anonymous> (test/functional/apps/monitoring/elasticsearch/overview_mb.js:26:7)
    at Object.apply (/dev/shm/workspace/parallel/13/kibana/node_modules/@kbn/test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)

and 1 more failures, only showing the first 3.

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
monitoring 725.7KB 728.0KB +2.4KB

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants